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Abstract 

This paper develops the theory of pushdown dimension and explores its relationship 

with finite-state dimension. Pushdown dimension is trivially bounded above by finite-state 

dimension for all sequences, since a pushdown gambler can simulate any finite-state gambler. 

We show that for every rational < d < 1, there exists a sequence with finite-state dimension 

d whose pushdown dimension is at most ^d. This establishes a quantitative analogue of 

the well-known fact that pushdown automata decide strictly more languages than finite 

■ automata. 
O ■ 

(N ; 

>v 1 Introduction 

a ; 

^ ' The dimension of a set of points was first explored by Hausdorff who showed that there 

j — ■ exist sets of points with fractional dimension, now termed fractals. Infinite sequences drawn 

from a finite alphabet can be viewed as points on the unit interval. Lutz ^01 showed that 
i j the Hausdorff dimension of a set of infinite sequences could be characterized by the rate at 

f-H ■ which money could be taken away from a gambler that is trying to make unbounded money by 

betting on all the sequences in the set. In other words, the higher the dimension of a set, the 
more random and unpredictable are its elements, and so the more difficult it is to make money 
betting on its elements. (A precise definition follows in later sections), 
■^j- ■ Though all singleton sets of sequences - i.e. all individual points - have Hausdorff dimension 

0, by restricting the computational power of the gambler, individual sequences can be assigned 
a non-zero dimension. Resource-bounded dimension (constructive dimension pushdown 
dimension 13 , finite-state dimension |2J, etc.) is a measure of the density of information or 
randomness in a sequence as it appears to a gambler whose computational power is limited by 
ly-} | the resource bound. Accordingly, the finite-state dimension of a sequence [2| is the degree to 

which the sequence appears random to a finite automaton, and the pushdown dimension of a 
sequence Ell is the degree to which the sequence appears random to a pushdown automaton. 

A finite-state gambler is a finite automaton that bets money on the next character according 
to its current state. A pushdown gambler is a finite-state gambler augmented with an infinite 
^ stack memory, and it is allowed to vary its state transition and its bet at each state depending on 

the character appearing at the top of the stack. Since any finite-state gambler can be simulated 
exactly by a pushdown gambler that makes no use of its stack, pushdown gamblers are at least 
as powerful as finite-state gamblers, and hence dimprj(S) < dimps^) for all sequences S. 

We show that there exist sequences with a pushdown dimension strictly less than their 
finite-state dimension. Specifically, for every rational < d < 1, there exists a sequence S with 
dimFs(S') = d such that dimprj(S') < \ dimp5(5). It is well known [H] that the class of languages 
decided by a pushdown automaton is a strict superset of the class of languages recognized by 
a finite automaton. The result presented here thus gives a quantitative, information-theoretic 
estimate of the extra power of a pushdown automaton over a finite automaton. 

Weighted finite automata, of which finite-state gamblers are a special case, have also been 
studied in other contexts • 

1 Department of Computer Science, Department of Bioinformatics and Computational Biology, Iowa State University, 
Ames, IA, 50011, USA. ddoty <at> iastate <dot> edu. This work was supported in part by National Science Foundation 
IGERT grant DGE-9972653. 

2 Department of Computer Science, Iowa State University, Ames, IA, 50011, USA. This work was supported in part by 
National Science Foundation Grants 9988483 and 0344187. 



^1" 



1 



Section [21 establishes preliminaries and notation, section |3] defines dimension, finite-state 
dimension, and pushdown dimension, section |1] establishes the separation of finite-state and 
pushdown dimension, and section El concludes and states open questions. The appendix contains 
proofs. 



2 Preliminaries 

We write Q for the set of all rational numbers, Z for the set of all integers, N for the set of all 
natural numbers, Z + for the set of all positive integers. Let logr = log 2 r. 

Let £ be a finite alphabet of characters. £* is the set of all finite strings drawn from S. The 
length of a string w E £* is denoted by \w\. A denotes the empty string. For I E N, £' denotes 
the set of all strings w E £* such that \w\ = I. w denotes the reverse of w. For w, y E £*, wy 
denotes the concatenation of w and y. For i > 1, w l denotes the string ww . . . vu . 

i times 

£°° is the set of all infinite sequences drawn from £. For S E £°° or £* and i,j E N, 
we write S[i] to denote the i'th character of S, with S[l] being the leftmost character, and we 
write S[i . . j] to denote the substring consisting of the i'th through j'th characters of S, with 
S[i . . j] = A if i > j. We write S n to denote S[l . . n], the n'th prefix of S. If n < 1, S n = A. For 
S E we write S[n . .] to denote S without its first n — 1 characters; i.e. S n -iS[n . .] = S. 

Let / E Z + , !o £ E' and S E We write #(w,S n ) to denote the number of times w 

appears as a substring of S n . Let the frequency of w in S n be defined freq(w, S n ) = ^zjx^p ■ 
Let the frequency of w in S be defined freq(w, S) = lim freq(w, S n ) = lim &( w ' Sn ^ ; when this 

n— »oo n— >oo 

limit exists. 

We state the following obvious lemma without proof, which states that adding a finite prefix 
to a sequence cannot alter the limiting frequency of any substring. 

Lemma 2.1. Let S E S°° and w,u E £*. T/ien, if freq(w,S) is defined, freq(w,S) = 
freq(w, uS). 

A sequence 5 E S°° is normal if, for every w; E £*, freq(w,S) = |£|~ l*"L In other words, S 
is normal if, for every string length I, all strings of length I occur with the same frequency. 

Note that given S n and / < n, freq(w, S n ) defines a probability measure on the set 
Accordingly, we can speak of the entropy of this probability distribution. Let the I 'th normalized 
entropy of S be denoted 

2 \ 

Hi{S) = — liminf V freq(w, S n ) log — . 

flog £ rt-oo ^ freq(w,b n ) 

Note that this exists even if freq(w,S) does not, since the liminf is being used. This is the 

n— >oo 

limiting entropy of the distribution of strings of length I in S, normalized by the term ^^rgi 
to fall between and 1. Thus, the more uniformly distributed are the strings of length I in S, 
the closer Hi(S) is to 1. Let the normalized entropy rate of S be denoted H(S) = lim Hi(S). 

l—*oo 

Likewise, the closer H(S) is to 1, the closer S is to normal, and H(S) = 1 S is normal. 



3 Dimension 



3.1 Hausdorff dimension 

The definition of dimension depends on the concept of martingales, which are strategies for 
gambling on character sequences, and s-gales (see which are martingales in which some 
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fraction of the winnings are removed after each bet. 
Let £ be an alphabet. 

Definition (s-gale and martingale) 

1. An s-gale is a function d^ : £* — > [0, oo) that satisfies d^ s \w) = Sae£ d(wa). 

2. A martingale is a 1-gale, denoted d(w). 

Intuitively, a martingale is a strategy for betting in the following game. The gambler starts 
with some initial amount of money (usually 1), termed capital, and it reads a sequence S of 
characters drawn from alphabet S. At each step, the gambler bets some fraction of its capital 
on each character in S. All of its money must be bet. Whichever character actually comes next 
in the sequence, it loses all the capital it bet on the other characters, and the fraction of its 
capital that was bet on the character that appeared is multiplied by The only restriction on 
a martingale is that the bet it makes on character S[i] of the sequence S should be a deterministic 
function only of the string Si-i, the characters that have appeared so far. The objective of a 
martingale is to make a lot of money, and it will make more money on a sequence if a larger 
fraction of capital is placed on the characters that actually occur in the sequence. All of the 
gambler's money must be bet, but it can "bet nothing" by betting ^ of its capital on each 
character in E. In this way, no matter what character comes next, its capital will not change. 

An s-gale is a martingale in which the amount of money the gambler bet on the character 
that occurred is multiplied by \T,\ S , as opposed to simply |E|, after each character. Note that 
s = 1 constitutes the original martingale condition. Since we will consider < s < 1, this will 
always constitute a reduction in capital from the capital the martingale makes. The lower the 
value of s, the faster money is taken away. Note that if a gambler's martingale for a string w is 
d(w), then its s-gale is given by d^ s \w) = \Y,\^ s ~ 1 ^ w ^d(w). 

Definition Let PCS*. P is a prefix set if no string in P is a proper prefix of any other string. 

Note that for any I G Z + , T} is a prefix set. The following generalization of the Kraft 
inequality was given in 

Lemma 3.1. Let s G [0, oo). If d^ is an s-gale and A C {0,1}* is a prefix set, then for all 
uG {0,1}*, Y.weA 2 ~ sHd{s) ( uw ) <d( s \u). 

Corollary 3.2. Let s G [0,oo). If d^ is an s-gale and A C {0, 1}* is a prefix set, then 

£ 2~ s \ w \d ( - s \w) < 1. 



Definition (s-success) 

Let S G We say that a gambler s-succeeds on S if the gambler's s-gale d^ satisfies 

lim supyj^^ n ) — oo. If the gambler 1-succeeds on S , we say it succeeds on S. 

So a gambler s-succeeds on S if it makes an unbounded amount of money betting on S, even 
with money taken away at rate s. 

Definition (dimension of a set) 

Let S C Then the Hausdorff dimension of § is 



dim H (S) = inf s s G [0, oo) 



3 an s-gale d that s-succeeds 
on every sequence S G S 
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3.2 Finite-state dimension 



Finite-state dimension is denned in analogy to Hausdorff dimension, where the gamblers imple- 
menting the s-gales are restricted to have finite-state computational power. In order to define 
finite-state dimension, then, we must first define finite-state gamblers, which were first studied 
by Feder jlj . Finite-state gamblers can be thought of as weighted finite automata |5] , where the 
weight on each state represents a percentage of money to bet on the next character. 

Let E be an alphabet. Let Aq(E) be the set of all rational probability measures over E. 

Definition (finite-state gambler 011]) 

A finite-state gambler (FSG) is a 5-tuple G = (Q,H,6, /?, go) where 

• Q is a finite set of states, 

• E is the finite input alphabet, 

• 5 : Q x T, — > Q U {_L} is the transition function, 

• (5 : Q — > Aq(E) is the betting function, 

• Qo £ Q is the start state. 

We write FSG to mean the set of all finite-state gamblers. 

If 5(q, a) = _L, for some q G Q and a G E, then that transition is undefined. We extend 5 to 
take strings as input with the function 5* : Q x E* — > Q defined by 

5*(q,\) = Q, 
S*(q,wa) = 5(5* (q,w), a). 

for all q G Q,w G E*, and o G E. 5* is then abbreviated 5 and 5(qo,w) is abbreviated 5(w). 
Intuitively, this allows us to identify 5(w) as "the state G is in after reading string w. n 



Definition (finite-state s-gale) 

Let G G FSG be a finite-state gambler. The s-gale for G is the function Oq : E* — > [0, oo) 



defined by 



dff(X) = 1 
dk\wa) = d G (w) • (3(5(w))(a) 



G 

for all s G [0, oo), w G E*, and a G E. 

Intuitively, the s-gale for an FSG G is determined as follows. An FSG G = (Q, E, 5, /3, qo) 
starts in state qo with initial capital 1. Assuming that after some time G has capital c and is in 
state q, the bet it makes on each character a G E is given by (3(q)(a). Assuming the character 
b appears next in the sequence, G then transitions to state 5(q,b), and its capital becomes 
c -/3(g)(6) -|E| S . 

Let G = (Q, E, 5, (3, qo) be an FSG. For q G Q, let d^) q be the s-gale for G if G is started in 
state q instead of qo. 

Definition (finite-state dimension) 

Let S G E°°. The finite-state dimension of S is 



dimp^S 1 ) = inf < s G [0, oo) 



(3G G FSG) lim sup d% ] (S n ) = oo 1 . 

n-^oo J 



4 



Thus, if s > dimFs(S), then there is a finite-state gambler G that s-succeeds on S, meaning 
G can make unlimited money betting on S, even if its winnings are multiplied by |S| S_1 after 
every character. 

Let X C E , and let S G Let dim^ (S) be the finite-state dimension of S when 

considered as a sequence drawn from alphabet even though it is really drawn from S. 

Lemma 3.3. Let S G and let £ C £'. T/ien dim£p (£) = dim^(S). 
3.3 Pushdown dimension 

Pushdown dimension 13 will be defined almost exactly as finite-state dimension, but the gam- 
blers will have an infinite stack memory, which will allow them to alter both their state transitions 
and their bets based on the symbol currently on top of the stack. 

Definition (pushdown gambler) 

A pushdown gambler (PDG) is a 7-tuple P = (Q, E, T, 5, (3, qo, z) where 

• Q is a finite set of states, 

• £ is the finite input alphabet, 

• r is the finite stack alphabet, 

• i : Q x T x (S U {A}) — > (Q x r*) U {_!_} is the transition function, 

• (3 : Q x r — > Aq(S) is the betting function, 

• qo £ Q is the start state, 

• z G r is the stack start symbol. 

We write PDG to mean the set of all pushdown gamblers. 

Note that the transition function 5 outputs a next state and a string w G T*. The top 
character is always popped and replaced with w. If a is the symbol currently on top of the 
stack, and P needs to add a character b to the top, it pushes the string ba. If it needs to leave 
the stack alone, it pushes the string a. If it needs to pop a character, it pushes the string A. 
Note that the strings are pushed in reverse order; the last character of the string is pushed first. 

Note also that the transition function 5 accepts A as an input character in addition to 
elements of S. P has the option not to read an input character and instead to alter the stack. 
To enforce determinism, we require at least one of the following hold for all q G Q and all a G T: 

• S(q,a,X) = _L, 

• 5(q, a,b) = _L for all b G £. 

The determinism condition requires that the PDG can't have the choice to read or 1 characters; 
the number of characters read is entirely a function of the state and the character at the top of 
the stack. 

We must also handle the special case that the stack start symbol gets popped. Since this 
represents the bottom of the stack, we restrict 5 so that z cannot be removed from the bottom. 
We restrict S so that, for every q G Q and a G {A}US, either 5(q, z, a) = _L, or S(q, z, a) = (q' , vz), 
where q' G Q and v G T*. 
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As before, if S(q,a,b) = _L for some q G Q,a G T, and 6 G {A} U £, then that transition is 
undefined. We extend 5 to the transition function 



where 5(q, a, b) = {Sq(q, a, b), 5r(q, a, b)). 5* is then abbreviated as 5. We then use the extended 
transition function 



in analogy to that used with finite-state gamblers, defined for all q G Q, a G T, v G F*, w G £*, 
and 6 G S by 



We then abbreviate 5** to 5, and <5(<7o, to 5(w). Informally, this allows us to use S(w) 
as shorthand for "the state and contents of the stack of the gambler P after reading string w" . 
We also extend (3 for convenience to the function 



(3* is then abbreviated (3. f3*(q,av)(b) means, informally, "The amount bet on character b when 
in state q, when the string av is on the stack." Note that only the top character a of av can 
affect any single bet, but for the purpose of examining multiple steps of the gambler, it may be 
necessary to keep track of what the entire contents of the stack are, since they may change from 
step to step. 

Pushdown s-gales and dimension are defined exactly as their finite-state versions. 

Definition (pushdown s-gale) 

Let P G PDG be a pushdown gambler. The s-gale for P is the function : S* — > [0, oo) 
defined by 

d { ;\\) = i, 

Jfiiwa) = d P {w) ■ (3(5{w)){a) ■ 

for all s G [0, oo), w G £*, and a G S. 

Intuitively, the s-gale for a PDG P is determined as follows. A PDG P = (Q, S, T, S, (3, qo, z) 
starts in state qo with initial capital 1. Assuming that after some time P has capital c, is in 
state q, and the character on the top of the stack is t, the bet it makes on each character a G S is 
given by (3(q, t)(a). Assuming the character b appears next in the sequence, if 5(q, t, b) = (q', w) 
G then transitions to state q', pops the top character off of the stack and replaces it with the 
string w, and its capital becomes c • (3(q, s)(b) ■ |S[ S . 




5** : Q x T+ x S* -> (Q x F*) U {±} 




(3* : Q x T+ -> Aq, 



defined for all g G Q, a G T, and u G T* by 



/3*(q,av) = (3(q,a). 
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Definition (pushdown dimension) 

Let S G The pushdown dimension of S is 

dim PD (,S) =inf Is G [0, oo) (3P G PDG) lim sup dfp (S n ) = oo 1 . 

^ n^oo J 

Thus, if s > dimp£)(S), then there is a pushdown gambler P that s-succeeds on S, meaning 
P can make unlimited money betting on S, even if its winnings are multiplied by |S| S_1 after 
every character. 

Pushdown gamblers are then nothing more than finite-state gamblers that make use of an 
unbounded stack memory, the top character of which can be used to inform the transition and 
betting functions. Additionally, PDG's are allowed to delay reading the next character of the 
input - they read A from the input - in order to alter the contents of their stack. During such 
a A-transition, the gambler's capital remains unchanged. 



4 Finite-state versus pushdown dimension 

4.1 Marker characters and finite-state dimension 

This section establishes that adding marker characters to a sequence, where the marker is not in 
the alphabet of the sequence, does not alter the finite-state dimension of the sequence, as long 
as the markers are spaced far enough apart. In other words, the addition of the markers cannot 
significantly hurt or help a finite-state gambler. 
Recall that, for S G 

H(S) = lim 1 liminf V freq(w, S n ) log ] — . 

i-oo/log S n^oo ^ freq(w,b n ) 

Let } 

H(S) = lim — -limsup V freq(w, S n ) log — . 

z— oo/logL n^oo ^ , Jreq(w,b n ) 

Lempel and Ziv ^H] showed that pfs(S) = H(S), where pfs(S) is the optimal compression 
ratio achievable by any finite-state compressor (see ^0] or j2] for a more complete description). 
Dai, et al. |2] showed that dimps(S') is identical to a slightly modified form of pfs(S)- A 
straightforward modification of the proof of Lempel and Ziv, combined with the result of Dai, 
et al., yields the following lemma relating finite-state dimension to entropy. 

Lemma 4.1. Let S G Then d\m FS {S) = H(S). 

Corollary 4.2. Let S G Then dimp^S") = 1 <^=> S is normal. 

Let S be an alphabet. Let S m = S U {m}, where m S is a marker character. Recall 
that dim^g (<S) is the finite-state dimension of S when considered as a sequence drawn from 
alphabet S m , even if it is really drawn from S C S m . 

Lemma 4.3. Let S G Let S' G SJ^ be constructed from S by inserting the character m 

immediately after the positions i% < 12 < 13 ■ ■ ■ in S such that the function f(j) = ij + i — ij is 
nondecreasing and unbounded. Then dim^^^S") = dim^T^S 1 ). 
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4.2 Bitstring characters and finite-state dimension 

In this section, we will interpret bitstrings of length I to be characters, the alphabet of the 
sequence will be a subset of {0, 1}' — {l'}, and the marker "character" will be lA An infinite 
binary sequence S G {0, 1}°° will then be simultaneously interpreted as an infinite sequence 
S G ^4°°, where A C {0, 1}'. In other words, every I bits of S will constitute 1 character from 
A. We interpret dim^' 1 ^(S') to be the finite-state dimension of S when viewed as an infinite 
binary sequence, and we interpret dim^ (S) to be the finite-state dimension of S when viewed 
as an infinite sequence drawn from A. 

Note that this interpretation of dim^g (S) is different from the meaning of dim^g (S) when 
{0,1} C A (i.e. in the sense of Lemma 13.3(1 . In the current case, the boundaries between 
characters actually change when moving from alphabet {0, 1} to alphabet A, in that a string of 
I bits is required to constitute one character of A. In the former case, for SCE' and S G 
dim^g (S) treats each character a G S in S as a character from We rely on context to 
distinguish these two scenarios. 

The following theorem establishes the relationship between the finite-state dimension of a 
binary sequence and its finite-state dimension when viewed as a sequence drawn from A C {0, 1}'. 

Theorem 4.4. Let I G Z+ and + A C {0,1}'. Then, for all S G A°°, dining' 1 " (S) = 
^dim^(S). 

4.3 Variations on the Champernowne sequence 

This section presents two variations on the Champernowne sequence ^ and shows them to be 
normal. 

First we need the following lemma, which establishes that splicing two normal sequences 
together results in a normal sequence, as long as the splicing takes increasingly longer substrings 
from each sequence. 

Lemma 4.5. Let S, T G S°° be normal over the alphabet S. Let 

U = S[1.. h}T[l . . h]S[h + 1 . . i 2 ]T[h + 1 . . i 2 ]S[i 2 + 1 • • i 3 ]T[i 2 + 1 . . i 3 ] . . . 

such that the function f(j) = — ij is nondecreasing and unbounded. Then U is normal over 
the alphabet S. 

Let d G (0, 1) n Q. Since d is rational, d = ^ for some d n , dd G Z + . Since d < 1, dd > 2. 
Since d > 0, d n > 1. Let / = d d and let A C {0, 1}' - {l'} such that log \A\ = d n . Note that 
\A\ = 2 dl , and that since d n > 1, \A\ > 2. 

Let oti G A* be the string consisting of all strings of length i over the alphabet A, concatenated 
in lexicographical ordering. Let c = l l . Define the sequences 

C = a\cT[a2(y. 2 a 2 a 2 azcty,a^aj,aj,a^ . . . 
C = aicaia 2 a 2 ca 2 lx 2 a^a^a^coi^a^a^ . . . 

Note that \cti\ = i\A\ l l \a\\ = i 2 \A\ % l. Champernowne and Merkel and Reimann |12j 

showed that the sequence ai«2 a ! ■ ■ ■ 1S normal over the alphabet A. 

Lemma 4.6. Let R = aTf a| a| . . . . Then R is normal over alphabet E. 
Lemma 4.7. C is normal over the alphabet S. 
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Note, however, that C and C are not normal to base 2, because no more than 21 — 2 l's 
appear consecutively in either sequence. In fact, they have dimension equal to d, as established 
by the following lemma. 

Lemma 4.8. dim^ { °' 1}) (C") = d. 

4.4 Pushdown gambling on a marked sequence 

The sequence C presented in section 14.31 has pushdown dimension bounded above by half of its 
finite state dimension. 

Lemma 4.9. dim^ { J 1}) (C") < \d. 

The main theorem of the paper follows and establishes that the pushdown dimension of C 
is bounded above by half of its finite-state dimension. 

Theorem 4.10. dim^ 1}) (C) < \ dim^ { °' 1}) (C')- 

Recall that d 6 (0, 1) HQ. Therefore Theorem 14. 1UI implies that for every rational < d < 1, 
there exists a sequence C with finite-state dimension d such that dimp£>(C") < ^dirnFs(C'). 

5 Conclusion 

We have shown that there exist sequences with pushdown dimension strictly less than their 
finite-state dimension. This was done by the addition of special marker strings that are placed 
increasingly far apart in the sequence. Because these marker strings do not occur in other parts 
of the sequence, the sequence is not normal, and this prevents our proof from showing that any 
normal sequence has pushdown dimension less than 1. The marker strings are needed for our 
proof, but it is not known whether they are essential to bound the pushdown dimension. It is 
possible that the original sequence, without the markers, has the same pushdown dimension. 

Nichols ^5] has shown that there is a normal sequence S such that a pushdown gambler can 
succeed on S, whereas the normality of S establishes that no finite-state gambler can succeed on 
S. However, the pushdown gambler fails to show that dimp^S) < 1, since the gambler makes 
money so slowly that it fails on S if any money is taken away at each step (i.e. if s < 1). 

Question Is there a normal sequence S such that dimp£>(S) < 1? 

We have shown that there exist sequences C such that dim PD (C') < idim FS (C"). The 
factor | seems artificial, and in our proof, it is an artifact of the particular pushdown gambler we 
designed. It is an open question whether this could be strengthened to show a larger separation 
between pushdown and finite-state dimension. 

Question Is there a sequence S such that dimp£>(5) < ^dimpsiS) 1 ? 

It is known that the dimension of an individual sequence with respect to some complex- 
ity classes can be equivalently expressed in terms of asymptotic compression ratios |2] and 
logarithmic- loss prediction [5]. Furthermore, finite-state dimension can be equivalently charac- 
terized in terms of entropy rates ^H] , and constructive dimension can be characterized in terms of 
constructive entropy rates 7 \. It would be useful to find other such alternative characterizations 
of the resource-bounded dimensions of sequences. 
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7 Appendix 



Proof ofLemma\EM We first show that &imfl){S) < £jH dim^(5). 

Let s > dim^(S'). Then there exists an FSG G = [Q,H,S, f3,qo) that s-succeeds on S. 
Construct the FSG G' = (Q', £', 5', ft, q' ) as follows 

• Q' = Q, 

5(q, a), if a G E 



5'(g,a) = 
/3'(g)(a) 



_L, otherwise ' 

(i{q) (a), ifaGE 
0, otherwise ' 



Since S" contains no characters from E' — E, for all n G N, 

/l£'l\ n 
dc(#0 = [\^\ d G (S n ). 

Let * = s §nif Then 

dg(5 n ) 4 |£'|(*- 1 )»d G ,( J S' n ) 

n \ n 



|E'|*\ n 



log [gj \ 1 



dG{Sn) 



m^ n d G (s n ) 



Thus G" i-succeeds on 5, since G s-succeeds on 5. Since this holds for every s > dim^(S'), 
M 

|S'| 



dimg } (5)<MM di 4 S 2(5) 



We next show that dim^^(5) > dim^(S'). 

Let t > dim)^ (S). Then there exists an FSG G = (Q, £', S, 0, qo) that ^-succeeds on 
S. Since S contains no characters from E' — E, assume without loss of generality that 
(3{q, a) = for all q G Q and all a G £' — E. This assumption can be made for the 
following reason. If a gambler does bet non-zero capital on a G E' — E, we can always 
construct a gambler that takes the capital G bets on a and uniformly distributes it to 
the remaining characters in E. Since a does not appear in S, this new gambler will make 
strictly more money than the old, and hence will s-succeed whenever the old gambler does. 

Then a straightforward reversal of the previous direction of the proof suffices to show that 
there is a gambler G' = {Q 1 ,S,5' , ft ,q' ) that s-succeeds on S, where s = t l °^ . This 

establishes that dim^^S) > wlxyl dim^g(5 l ). 
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□ 

Proof of Lemma \4.y\ Let S and S' be as in the statement of the lemma. Let I € Z + , and let 
w € T, 1 . 

Let there be k n insertions of the marker character m in S n (i.e. the insertion indices satisfy 
I < h < 12 < ■ ■ ■ < ik n < n < ik n+ \). Then S' n+k is the prefix of S' "corresponding" to S n . 
Note that freq(m, S' n+k J = ^ . 

Since — ij is non-decreasing and unbounded, (Vp G N)(3n p ) such that all markers after 
position n p are at least p characters apart. Hence freq(m, S'[n p . .]) < -. By Lemma 12.11 
freqim, S') < -. Since this holds for all p G N, freq(m, S') = 0. Since freq(m, S' n+kn ) = n ^ k , 
then fc n = o(n); fc n grows strictly slower than n. 

Since there are fc n occurrences of m in S' n+k , there are fc n (Z — 1) substrings of length I in S n 
that could have been changed by having an m inserted into them. In the worst case, every one 
of these substrings was our chosen string w. Thus 



#(w,S' n+k J > #(w,S n ) 



k n (l - 1) 



(1) 



# of w in S' n+k # of u> in S n # of w in £ n that 

could have changed 



Since w & T, 1 , it does not contain an m. Adding m's to 5 cannot add more w's to S. Thus 



Recall that k n = o{n). Thus 
lim (freq(w, S' n+k J - freq(w, S n )) 



and 



lim (freq(w, S' n+k J - freq(w, S n )) 



> 



< 



lim 

n->oo \n + k n — I + 1 n — I + 1 

#(w,S n )—kn(l-l) #(w,S n 



lim 

n— >oo 

lim 

n— >oo 



n + k n — I + 1 n — I + 1 

#(w,S n )-k n (l-l) #(w,S n ) 



n-l + 1 

k n (i-iy 



n 



l+l 



lim 

n— >oo V n — t + 1 



0, since k n = o(n) 



lim 

n— >oo \n + fe 



1 + 1 

#(w,S n ) 



lim 

n^oo \ n + K n — t + 1 



n - / + 1 

#{w,S n ) 



lim 



re - Z + 1 
#(w,S n ) #(w,S n ) 



(2) 



inequality Q 

since A; n = o(n) 



n— >oo I n-l + 1 n — I + 1 



inequality @ 
since fc n = o(n) 



Thus 



lim (freq(w, S' Ti 



n+k n 



freq(w,S n )) = 
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This establishes that freq(w, S n ) and freq(w, S' n+kn ) approach each other as n — > oo, for all 
w G E'. Let u/ G E^ — E f . Then freq(w',S n ) = for all n, since no m's appear in 5. Since 
freqim, S') = 0, 



freq(w ,S) = lim 

n.^oo 77, 

l#(m,S' ri 



< lim 



n— >oo re 
= Z • freq(m, S ) 

= o, 

where the inequality follows from the fact that for each m that appears in S' n , at most I substrings 
of length I in S' n could have that m in them, and hence belong to Y} m — E\ By the non-negativity 
of freq, freq(w', S') = = freq(w', S), implying 

lim (freq(w',S n ) - freq(w',S' n+k )) = 

n—>oo 

for all w' G T} m — Hence, 

(Vw G E^J lkn^ (freq(w, S' n+k J - freq(w, S n )) = (3) 

Thus 

Hi(S') = - — \— — liminf V freq(w, S' n ) log 1 

/log E m rw°o ^ freq{w,S' n ) 

= n — 7^7 liminf f r eq(w,S' n+k J log- p— = r 

Zlog|E m | n-oo ^ +« /reg(w;,5; +fcn ) 

= n — -R^T limmf ^2 freq(w,S n ) log- 1 by© 

Zlog £ m n-»oo ^— ' freq(w,S n ) 

= Ht(S). 

Since this holds for all I, H(S) = H(S'). By Lemma IP dim^ 5 m) (5) = dim^ s m) (S')- 
Proof of Theorem \4-4\ 



□ 



We first show that dim$j ,1|J (5) > dimfj(S). 

This holds trivially if \A\ = 1, so assume \A\ > 2. Let s G [0, oo) D Q such that s > 
dim F S (S). 

By our choice of s, there exists an FSG G = (Q, {0, 1}, S, (3, qo) such that G s— succeeds 
on S. Construct and FSG G' = (Q', E', 6', /3', q' Q ) as follows: 

• Q' = Q. 

• E' = A 

• for all q G Q' and u)£i, 

<5'(<?,w) = %,«;). 
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for all q € Q' and w G A, 



where 



and 



(3\ q )( w ) = { ttt' ^(A)^ 
1 0, ifB(q)(A)=0. 



B(q)(w) =f[p(6(Q,Wi-i))(w[{\ 

i=l 



Note that for all q £ Q', d,Qi %q is a martingale, and that A C {0, 1}' is a prefix set. Let 
q E Q'. Then 

B(q)(A) 4 



weA 

i 

■weA i=i 

w£A 

< 1. by corollary 13.21 

B(q)(A) < 1 (4) 



So 

for all q GQ'. 

Let u; € A and let q & Q'. Then 

da'q(w) 



B(q)(w) 
B(q)(A) 
> B(q)(w) by © 

i=i 

= d Gtq (w). 



So by induction, for all z £ A* , 



d' G (z) > d G {z). 
Let z € A* and u; € ^4, and let q = Then 

= 2'B( 9 )(u;)£fc(a!) 
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and so 



d G '(zw) 



\A\^M dQl{z) 



> \A\ 



B(q)(A) 
B(q)(w) 



1 



B(q)(A) 2 l B(q)(w 



-d G {zw) 



2 l B(q)(A) 



d G (zw) 



> —]-dc{zw). inequality 
2' 



So by induction 



Let t = EfpT Then 



d G ,(z) > 



2' 



d G (z). 



ki 

> |^4.|C*— i) ^ f\^Pj 1 d G (z) inequality © 



t< 

HZ 

2l 2 l 
|^|i^ 



> 



2\ z \ 
2«|«| 



<fe(*) 



d G (z) since |A| > 2 



> gjij-dGCO since \A\ > 2 
= 2( s - 1 )l z ld G (z) 

= 4 S) (*)- 



Thus G" t-succeeds whenever G s-succeeds. This establishes that 

•dim^(S). 



dim(g' 1}) (S) > £dim^(5) = ^^)< 



We next show that dim^ { °' 1}) (5) < dim^(S). 

Let s G [0, oo) n Q such that s > dim^(5), and let t 
,({0,1}), 



jjogJAj 



(5) 



Then it suffices to show 



that dining' (S) < t. By our choice of s, there exists an FSG G = (Q, A, 5, (3, go) such 
that G s-succeeds on S. 

Let ppref(A) be the set of all proper prefixes of the strings in A. Construct the FSG 
G' = (Q',Y,',5',/3',q' ) as follows: 

• Q' = Q x ppref(A). 

• E' = {0,1}. 
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• for all q G Q', w G ppref(A), and b G {0, 1}. 

{(q,wb), if wb £ ppref(A); 

(S(q,wb),X), if wbe A; 
_L, otherwise. 

• for all q eQ', w e ppref(A), and 6 G {0, 1}, 

[ 0, if B(g,w) = 0. 

where 

B(g,t/>) = J] P(q)(wu) 
ueA(w) 

and 

A(u>) = {u G {0,1}* | urn G A}. 

• q'o = (go, A). 

In the non-degenerate case (where B(q,w) > 0) 

B(q,w0) + B(q,wl) 



P'(q,w)(0)+P'(q,w)(l) 



B(q,w) 



For all w G ppref(A), A(w) is the disjoint union of A(w0) and ^4(w;l). So B(q,w0) + 
=B(q,w). Therefore ^'(g, iw)(0) + /?'(g, w)(l) = 1. 

Note that for all q G Q, B(q, w) < 1 for all w; G ppref(A) U A This follows from the fact 
that w = X maximizes B(q,w). B(q, A) = Yl P(q)(w) = 1, by the constraint that /3(g) is 

a probability measure over A. 

Intuitively, G"'s martingale bets I times every / bits, in such a way that the I bets made 
will telescope to simulate the bet made once every I bits by G. 

Let z G A* , w G A, and g = <5(z). Then 



i=i 



2^)11 ^ 



x B{q,Wi-i) 

B(q,X) 
> 2 l d G ,(z)B(q,w) 
= 2 l d G ,(z)P(q)(w), 

and 

d G (zw) = \A\(3(q)(w)d G (z). 
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So by induction 



and 



So 



Thus 



i 



d G ,(z)>H&P(5{zi l ))(wi 

i=l 



d G (z)=H\A\P(8(z tl ))(w). 



i=l 



dc(z) 
d G (z) 



> 



n 2 l P(8(z u ))(w) 

i=l 



n \A\P(5(z u ))(w) 

i=l 

. . . M 
2* 



da>(z) > [jj] d G {z). 



d%]{z) ^ 2 (*- 1 )l^d G ,(z) 



2' \ 



= 2*i z i|^rT 1 (i G (z) 

= 2 t l*l~¥ log l A l(i G (z) 

s log I A I i i I z I n i4i 

= 2^M 2 l-V lo sl A ld G (z) 

= 4 S) (*)- 

Therefore G" i-succeeds when G s-succeeds. This establishes that 

dim«J 1}) (S) < £ dim$(S) = ^Mdim^(5). 



□ 



Proof of Lemma \4-5\ Let n = ij, for some j G Z + . Let k n = j. Intuitively, k n is the number of 
splices taken from S n and T n apiece to form Z2n- Since ij+i—ij is nondecreasing and unbounded, 
lim %• = 0. 

Let I G Z + , and let w G E . Then freq(w,S) = freq(w,T) = Because there are 

only fe n places in S n at which it was "broken" to be spliced into T n , at most k n (l — 1) instances 
of w in S n could have been disrupted by the splicing and hence not appear in Z2 n - The same 
argument applies to instances of ro in T„. Thus 

#K Z2n) > S n ) + #( W , T n ) - 2k n (l - 1) 
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Therefore 



jreq(w,z) = lim 



n 



= lim 

n^oo 2n 

^ #KS n ) + #KT n )-2fc n (J-l) 
> lim 

n^oo 2n 

= 1 lim #(*>>Sn) + 1 Um #Kr w ) _ Um fcn 

2 n^oo 71 2 n^oo 77, n— »oo 71 

1 ,. #(w,5n) , 1 ,. #(w,T n ) 
— lim h — lim 



2 n^oo 77 2 n^oo 77 

- ^freq(w,S) + ^freq(w,T) 

This holds for all w S T} . ^ freq(w,z) = 1, so, for all w G S', freq(w,z) = 
Since this holds for all / £ Z + , z is normal. 



□ 



Proof of Lemma fjl] Let =0102 a| . . . a|. Let = |i?(A;)|, so that R nk = Let 

U = aia^a^ . . .. 

Let I € Z + , and let to G S . Then freq(w,U) = |E|~ , by the normality of £/. For every 
instance of w in x that does not cross a boundary between a\ and a£j^, then a corresponding 

instance of w appears in R, since R = aTf a| • • •• There are at most (fc— 1)(Z — 1) instances of io 
that could lie across such a boundary. Since f(k) = rik+i — n& is nondecreasing and unbounded, 
lim = 0. 

Therefore, for all k £ Z + and all w E S' 

#{w,R nk ) > #(w, U nk ) — (k — — 1) 

So 

freq(w,R) = lim ^ "' 



n— >oo 77, 



> lim 



k— >oo 77/% 

#( w ,U nk )-(k-l)(l-l) 



lim #(^„J _ lim (*-i)(i-D 

fc-^oo 77fc k^oo nk 

lim *("^"'> 

A:— >oo 77^ 

lim #KCW 

n^oo 77, 

freq(w, U) 
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Since this holds for the reverse of every string in T, 1 , it holds for all w € T, 1 . Since 

freq(w,R) = 1, 

freq(w,R) = for all w G T, 1 . 

Since this holds for all / € Z + , R is normal. 

□ 



Proof of Lemma \4- 7| This follows immediately from Lemmas 14.61 and 14.51 and the normality of 

□ 



Proof of Lemma \4 ■ y\ Let s > s' > d. It suffices to show that dim^ 1 ^ (C) < |s. Let A c = 
Au{c}. ' "' ' _ 

We construct a PDG P that does the following. It reads the sequence C = aicala\ca\ . . . 
in two alternating stages. The first stage involves reading the substring a-c, and the second 
stage involves reading the substring a\. In the first stage, P bets optimally for any FSG, while 
the bits it reads are pushed onto the stack. Once c has been read, P pops c from the stack, and 
then uses the string it pushed, which is a\, to bet optimally on the string that follows, which is 
a\. It pops bits until the stack is empty, at which point ct^\ follows, and the gambler begins 
again. 

As P is pushing bits onto its stack, it bets an equal amount (dp (a) = on all bitstrings 

a € A. It bets a small amount {dp{c) = 2 l e) on the bitstring c = l', and this bet can be made 
vanishingly smaller by shrinking e, although some positive bet must be made so P's capital 
does not become when it encounters c. The requirement that e < 1 — 2 l ( s '~ s ^ ensures that P 
[2 s ) -succeeds on C", which is shown formally below. P bets nothing on any bitstring a A c . 

Thus, P's martingale behaves optimally for any finite-state martingale when reading the sub- 
sequence a\cc3,c . . . a\c . . ., and it doubles its money on every bit when reading the subsequence 

aTa! • • • a l 

Formally, the PDG P = (Q', £', V, 5', f3', q' , z) is defined as follows on input S G {0, 1}°°: 
P{C) 

1 i «— 1 [> current bit of C 

2 while true > each iteration k reads Q^cajjj: 

3 do repeat D> push bits until marker found 

4 w <— A 

5 for j <— 1 to I D> set w to next block of length I 

6 do bet according to (3{w) on C'[i] 

7 w <— wx[i] 

8 push C'[i] onto stack 
9 

10 until w = 1 

11 pop I bits from stack 

12 while stack is not empty 

13 do bet all capital on bit on top of stack 

14 read C'[i] 

15 i^i + l 

16 pop 1 bit from stack 
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where 

[ 0, otherwise. 
B{ w ) = ^ B(wu) 

u<=A c (w) 

A c (w) = {u G {0, 1}* | TO G A:} 

B(a) = { W' ** eAi 
{ e, if a = c; 

0<e < l-^"' - "). 

Note that, for all a <G A c , 

i 

d P {a) = 2'H0(oi-i)(a[*]) 



B(oi) 



i=i 

ti V^K-i), 

B(X) 

; MgA e (a) 

£ B(Au) 

«ga c (a) 

of 

E *(«) 

2 / B(a) 



Thus, for all a G i, 



and, for c = l l 



= 2'B(a). 
d P (c) = 2 z e. 



Recall that dp(c) = 2 l e, and that P makes the same capital (dp(a) = on each 
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"character" a € A 



and 



Thus, 



d P {a k k ) 



d P {a k k ca k k ) 



(d P (a k )) k 

dp(^a)* of^Tngs # ° f charactcrs P cr strin S 



V 



= 2 



\A\ k k 2 



= 2 K 



\A\ J 

\A\ k l f 1 ~ e \ 

\A\ J 



k 2 \A\ k 



d P (a k ) = 2NI 



2 k 2 \A\ k l 



^k 2 \A\ k l 
e 22k 2 \A\ k l+l 



1 -e 



k 2 \A\ k ^ 



\A\ 



(2 
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Let t = ±s. Then 



d%\a k k cal) = 2 ^\<<W k2 \ A \ kl+l 

2(t-l)(2k 2 \A\ k l+l) e 2^k 2 \A\ k l+l 
'1 _ e \k 2 \A\ k 



k 2 \A\ k 
l_ e \ k2 \ A 



\A\ 



e 2t(2k 2 \A\ k l+l) 
2tl e 2t2k 2 \A\ k l 



\A\ ) 
1 - e 



k 2 \A\ k 



2 tl e2 t2k 2 \A\ k l ( (1 ~ e)tl \ 

{ \A\h J 

/ „ .±^tlk 2 \A\ k 



tlk 2 \A\ k 



\A\ti 

2 ;I < I 2 "' 



ti I o2 ( 1 - e J" 



1 x tlk 2 \A\ k 



(2 dl )Tl 



j_ \ tlk 2 \A\ k 

> 2"e I 2 2 " ~ " 



2 (l-e) « 



{2 s ' l )u J 
= 2M2 2(1 - e) " I 



= 2" e (2 2 -T(l- e )ii) 
= 2" e (2 2 -^(l- e )n) 
= 2«6(4 1 -T(l- € )^) 



Recall that e < 1 — 2^ s ' s ). Then the term in the parentheses, 

4 i-4(l_ e )§ > 4^(1- (l-^-))) 1 



= 4 1 -T4i( s '- S ) 
= 1. 



Thus d ( Q(a k ,ca k : ) grows without bound as k — > 00, whence P t-succeeds on C". Therefore 
dim^CCO^.^dim^^)^^- 



□ 
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